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Introduction 

— — . — — — — . ^ — . ^ ^ 

Several" researchers have emphasised the importrance of the relationships 
between program evaluators and key decisxon-^makers for achieving substantial 
utilization of evaluation findings (Co:^^ 1977^ Holley^ 1978, Patton, 1978). 
After reviewing' the literrature on use of evaluation information^ Cox concludes 
that "improving utilization is not going to be so much a matter of training 
tnanagers in the subtleties of i;esearch methodology and interpretatioti as it 
will be training evaluators in organizational and political realities and 
communication skills and having them placed in organizations so that they are 
in extensive contact with -relative administrators" (Cox, 1977^ p. 508), Our * 
own survey of special education program staff "memhers supports tox's conclusion^ 
highlighting 'the need for evaluators to develop "an increased understanding 
of special education staff programs, and constraints." (Drezek, Note 1) 

Hypot^hesis 

Somewhat ^less has been said, however^ on how the personality or managerial 

style of Che decision-maker effects utilization of the data. We hypothesize 

that inareased inteTperBonal involvement betb^esn the evaluatop and the manager' 

ikey deaision-makBv) lead to inereaBed impact fend utzHzation of eval-' 

♦ 

nation f%7idings and that the strength of this effect will depend upon 'the 
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managBT^s "style,'^ That is> we believe the effects of interpersonal involve^" 
'ment will be of less importanc^^or managers *who already creatively generate - 

r 

and use data than fc^ managers whose efforts are directfed primarily toward 
program survival or toward the enhancement of the, program' s image, 

Thi$ paper represents our initial exploratory phase in which we attempted 
to clarify our conceptual framework and de^^elop instruments to measurjp the 
key variables— interpersonal involvement, impact of * evaluation, and managerial 
^tyle* ^ ^ . ' -^^ 

Int;erp^rsonal Involvemenlt - 

We selected two potential wa^^s of measuring in terpersronal 'involvement * One 
method is the use of a task coda (see Attachment 1) which will indicate the ' , 
proportion of time an evaluator spends vis-a-vis with project staff for. each 
project, 

Our second approach to uleasuring interpersonal involvement is to ask th^ 
manager as to^he extent he was^ involved in, formulating evaluation questions^ 
interpreting findings, and generating recommendations. Two five point*^ items 
were devised "^f or this (see Attachment 2, part I) • — 

In addition to ^cooperatively developing and carrj'ing out an evMuation 
plan and going over findings, inter|>ersonal involvement contains other more 
qualitative aspects... Among those aspects are: ^ ^ ^/ 

(1) Developing an interest in the project — discovering the manager^s goal.s 
feelings, and values. ^ . 

* (2X Sharing your perspective— letting the manager know hoi^ you feel 
and think* 

(3) ' Developing mutual respect— recognizing each other^s strengths ^and 

weaknesses and allowing for differences in opinion. 

(4) v Attempting to meet each other's needs end compromising when 

'necessary. I 



* Impact of Evaluation 

f 

At our education service center the evaluation plan consists of * 
^ formulating several questions ^nd producing a .s^ies of doc^uments to answer 

^ these questions^ Therefore^ we decided that one way to assess impact would ^ >^ 

be to have an independent person ask managers about the Impact of three evaluation 
documents chosen at random* Six areas of potential impact ^ere chosen for pro-- 
bing by an independent consultant. Managers would-be asked as to whether the 
document (1) changed the way services are prpvided, (2) changed program management ^ 
^ (3) led to a reallocation of resources, (4) influenced the .selti^^»lon of goals 

and objectivear, (5) l^d to new ideas, or (6) had any other form of impact Managers 
arfe asked to state specific instances for each area of impact (see Attachment, 
part II)* 

* « 
Managerial Style * " ' - * ^ 

The most difficult but probabj^ the n\ost creative aspect of our task was 
to define managerial style in a marmer relevant to the manager *s probable use J 
^ of evaluation data* We looked at how managers processed information from 

three {Perspectives (1) primary purpose of actions,^ (2) basis for actions^^and 

(3) us§ of ^ information. We loosely related managerial 3tyle for processing 

information to an individual's ^^'st age of development ranging from^ a position 

of insecure dependencjwJto creative 'self-actualization* With, r^pect 

to managerial style ♦this controuum divided itself into five categories which- ^ 

we labeled "tactical,^* "political **cybernetical/' '^analytical/' and 

''experimental/' * • * ' ^ ^ 

A manager who^ is in the first category-*-**tactical''— is one whose prime 
focus is ensuring^ the surylval of his/her program. A manager whose efforts 
are directed toward gaining acceptance and enhancing the image of his/her 
program ve label ''political." The manager who seeks aehievement within the 
limited framework of the existing administrative structure we label ^'cyber- 
, netical/' and the manager who seeks to achieve goals beyond those established 
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within, the existing administrative structure we lab^ "analytical." Finally, 
the self -actualized^ creative person is represented by the >«^'experimental'* 
manager, who actively manipulates his/her environment. From this continuum ^ 

we developed a rough measure of ''managerial style for processing information/- 

r 

(See Attachment 3). 



Other Considerations 

As a secondary hypothesis we predict that In actual fact evaluators tend 
to spend ^^s time with manager^ who score low on our scale of managerial 
style^ — those people generally resistant to the whode process^ of * evaluation. 
Although bypassing these mana||rs may make it easier for the evaluator to 



produ<!e a document containing Ivaluation findings » this practice neglects the 
I 

real objective of evaluation— ytili^ation of information leading to program 
improvement. Moreover, this practice causes evaluators to spend less^ time 
with the AJtery people who, in out opinion, should receive more ^attention. We 
hypothesize that the best way to deal .productively with "tactical", of "poli- 
tical" managers is to fully involve them in all phases of 'the evaluation 
process — even though this strategy may require' that the evaluation plan 
initially be limited to only a fe^ crucial areas of ^ the program. 

Note that' the behavior of any particular manager fluctuates, A manager 
at one point in time or in regards to a specific issue may respond^ with a style 
' that is not his/her usual* - Furthermore^ a manager ' s- style is' likely to change 
the person becomes more familiar vith the evaluator and the evaluation process. 
Typically, the beginning of a manager-evaluatror relationship requires extensive 
sharing of information and developing rapport. 

One DOssible Tneans of testing our hypothesis would be to use a jnanagef 
as his/her* own control and do a before and after study with the intervention 
being fncreased interpersonal Involvement. However, if the hypothesis is 
tested across managers &s is our intention, then managers must be matched 
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according to^'^style'* as the probability of greate^ utilisation of , evaluation 
findings. by ^^experimental" and ''analytical" managers than ''politicar' or 
"tactical"^ ones .is built into our ^measure of managerial style. 

Summary ' . 

In .conclusion, we have formulated an aval ua tor-manager interaction model 
for predicting impact <?f evaluation. We have developed tentative Instruments 
for measuring the variables of in,terpersonal involvement. Impact *of evaluation/ 
and managerial style. We can now test our hypothesis that increased inter- 
person^ involvement between evaluator ^^^|^d manager ^111 lead to increased use 
of ^valuation data with managerial style 6ts "a moderator variable; that is^ 
the extent of involvement between evaluator and manager will be of particular 
importance for ^managers rated "tactical'' or "political," If our conceptua- 
lization is correct, evaluators can improve their efficiency and Impact by 
shifting the bulk of th^ir interpersonal inwlyement^^way from managers wljo 
^ should be given the data to use independently to managers who are more ^ ' , 
reluctant tp use. evaluation data to change their programs. » 
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Attachment' 1 



TASr COBE^CATAGORIES 



TASK 
CODE 



1 KEEPS ASSESSMENT 
Consultation with Center personnel 
desiring data* 

1#2 File search for previouBly obtained 

data* ^ ^ 

1.3 Preparation of data collecting in- 

atrumei^ts^ 
1^4 .Selecting sample, collecting, data • 
1^5 Tabulating, processing & analyzing 

results^ i 
iTl^ Report preparation 
*1^7 Consultations to interpret & ex- ^ 

plain *f indings. 
1^8 Administrative & Supervising 

CHARGEABLE : ^ Xt .r*lated to 
. . ^ funded project 

iJON-CHARGiEABLE : If a new proposal. 

' . ^ i 

2 PROPOSAL DEVELOPMENT ^ * 



:eF( 



TASK 
CODE 



*2.1 Conferences, tneetings, & discussions 
2.2 Reviewing , objectives ^ 
2*3 Preparing ^the evaluatibn section. 

2.4 Reading or reviewing proposals for 
comment* 

2.5 Writing complete proposal 
2*6 Administrative & Supervising 

CHARGEABLE : If a continuation 

proposal • ^ 
NON-CHARGEABLE : If a new proposal 

3 EVALUATION PLANNING 

.3*1 Consultation within evaluation st^ff 
3.2 Preparing the Evaluation Plan 
CHARGEABLE : If related i to 

^ funded project 

4 DATA COLLECTION 



4.5 Administering exams, tests, question- 
naires, etc. 

4*6 Distributing & collecting question- 
naires 

4*7 Briefings pertaining to data collection 
4*8 Administrative & Supervising , 

CHARGEABLE : If related to funded 
project 

' 5.^ DATA PROCESSING ^ ^ ^ 

5*1 Developing a record keeping system 
* 5.2 Processing workshop evaluation forms. 

5.3 Tabulating or scoring other' instrument 
dal:a» , 

5.4 Preparation for card punching h compu*- 
ter run. ^ ^ 

5.5 Statistical analysis of data 

5.6 Reviewing .records, reports, or other^ 
^ Information. 

5*7 Administrative & Supervising 

CHARGEABLE : If related to funded 
project 

6 REPORTING EVALUATION 
6.1 Writing findings, memoranda, interim, 

or final reports* 
*6.2 Disseminating evaluation results 

6.3 Administrative & Supervising 
, CHARGEABLE ; If related to funded 

project 

7 TECHNICAL ASSISTANCE 
7.1 Travel time 'connected with providing 

assistance. * 
7^2 Conducting or participating in 

, workshops 
7*3 Other assistance provided 

7.4 Administrative & Supervising 
CHARGEABLE I If related to a 

funded project 



4*1 Developing or selecting Instruments 
4*2 Travel time connected with collecting 
data. 

*4*3 Conducting interviews; telephone or 
in-persAn* 

* ObseivvSlftons; workshops, schools, # 
]^|^(^ centers, etc* * f>etvf^wc. p^^ncr /^m^o/o^ ^j*r^ ^nt^^^e^ ^r-^^^ 



TASK 
CODE 



8 INTERACTION WITH PROJECT STAFF 
8.1 Meetings involving learning 

about program 
8^2 Contact about developing evalua- 
tion plan 

8.3 Briefings, conferences, & sign-" 

off of plans 
8. ,4 Verbally conveying f indings or 

results of evaluation 

8.5 Providing technical consultation 

8.6 Formulating recommendations 

8.7 Consultations involving using 
evaluation data. 

8\B Administrative & Supervising 
^ CHARGEABLE : If plated to a 

f-u«ded pr b j ec t 



9.1 
9.2 
9.3 
9.4 
9.5 

9.6 

9.7 



10 

10.1 

10.2 

10.3 

10.4 

10.5 

10.6 



INTBRN/VL PROCEDURES 

All Inservice sessions/^ retreat 

Center, component, & unit meetings 

Area conferences & workshops 

Special studies & projects 

Preparation of professional 

papers 

Screening applicants fpr em- 
^Ijloymeiit 

Attendance at professional 
meetings ^ , . 

Administrative & Supervising 
CHARGEABLE : If related to a 

funded project. ^ 
Prorated for per- 
^ sonnel^wlth^ split 
funding 

ALL ABSENCES 

Vacation 

Sick 

Per sonal 
Jury/military 
ESC--20 Holidays 
Other 

CHARGEABLE : Prorated fol: person- 
nel with split funding. 
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t:mpact of evaluation ^services 



By identifying the type of services which, ledd to impact^ 
we may increase the future impact pf Evaluation Services, 



Instruction: \h^ following three documents represent all or a select, 
^ sample of the work Evaluation Services has done for your 
^ « program* I will let you review each one of these one at 

a tinxe so ,you can refresh your memory. Then I will ask 
you some questions regarding their impact on your program. 



Document Tittle: 
Pro gram : 



Person Interviewed 



Date of Document 




^ 1, -Involvement 

J, 

1) How involved were you at the beginning of this work such as choosing 
the subject, formulating .tAe questions, selecting items, etc.? ^ * 

'1 = not involved 2 » limited 3 » moderate 4 • considerable 5 « extensive 

involve- 'involve- involvement involvement 

ment ^tnent 

* • . * * ^ 

2) How involved were you during and at the completion, of this work 
such a$ receiving feedback; discussing findings, going over recom^ 
tfendations, ect;"? ' ^ 



1 ^ not involved 2 ^ limited ^3 * moderate 4^* considerable 5 = extensive ^ 

involve-" involve-^ involvement * involvement^ 
ment ment ^ 



II. Impact 



\at:e the extent ^of impact this document has had in eachvof the following areas 
using the code: 

1 « no impact 2 = limited ' 3 moderate 4 « considerable 5 - extensive 



Code 



1 2 3 4 5 



impact 



impact 



impact 



impact 



(1) Led to changes in the way services are provided. 
Specify: 
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Code 



12 3 4 



V 



5 (2) Led to changes in the^wky the program is managed 
Specify: 



12 3 4 5 (3) Led to a reallocation of resources (personnel time^ monies, ma- 
terials) ♦ . , » 

* specify: . 



1 2 3 4 5 ^ (A) Influenced the ,selet:tldn of goals .and oblectives or the assign-- 

ment of program priorities. ^ y 

Specify: . ^ , nir ^ , 5^ . 



1^ 3 4 5 (5) Led to or provided new ideas' or ways of looking at the program* 
Specify: ^ - 



1 2 3 4 5 (6) What^ other wa^s did this document and -the in 

to it have ari impact on this .program. 

/ Specify: 




t^^s relating /§->v^# 



ISeotions X. and II. are repeated for tw6 add^itionat doowrientTj 



111. In addition to the documents we have discussed, can you specify any other 
ways Evaluation Services has^ had a ^'considerable'* or ^'extensive" impact 
on your program? . ^ r ^ . - 
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y • . Attachitfent 3 *v 

MANAGEKIAL STYLE ^or Processing Infoimatlon 

Purpose: ^ To determine the type of managers you work with, * / 

I » * 

i; Name: ^ \ ' B^rogram or Project f 



. Manage^:: 



II. *Read,the descriptions in the^ boxes for each of the three areas Then note the category most descripti^^e of the 
above named 'manager '^and assign a number, ^ P<on may use decimals if .you like)/ ^ ♦ ^ 



i) Primary 

pi^rpose of 
actions 



Rating: 



2) 



Basis 

for action 



Rating: 



5*0,, 
Experimental., 



4.5 - 



Analytical 



1.5 - 



3) Use of^ 

Information 
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iprng:^ 



Find the "best" or. 
"elegant" vay to 
satisfy the needs , 
for which jthe , * 
program was set^ 
up within -the 
lllsoadest framework • 



Find a "good" ^ 
way 'to satisfy 
the needs for^ • 
which the program 
was set up 
with in \a broad 
framework/'* 



Cybej;netical 

He^et program ^ 
goal/ objectives 
within a limited 
framework* 



Political 



1.0 
Tactical 



Make th^ program 
"lopk good" to 
a host of 
external parties* 



/ 



Generates, 
considers, and 
tests pirogram^ 
options* 



Generates and 



^1 



considers program { information to 



Responds to new 



options * 



make program 
changes . 



{ Responds, to, 
1 problem^'JM^mands 
I ' to mafe,e^ program 
\ changes. 



Generates, 
collects, and 
s ys tema t ically 
organizes a 
variety of 
Information* 



Collects and 
systematically 
organizes a 
variety of 
inf 9rmat;^pp* 



Sy s||ema 1 1 c a 1 1 y 
organizes 
incomij^k 
Inf orij^Ron • 



Jnsure the 
' continuation/ 
survival of the 
pVograift. 



Reacts to 
unsettling 
elements in an 
effort to 
maintain* the 
existing 
structure. 



Collects and 
organises 
"required" or 
^^requested" 
information* 



Maintains 
limited and 
"select" 
information to 
support position. 
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